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This work develops a new modular architecture that emulates a recently- 
discovered biological paradigm. It originates from the human brain where the 
information flows along two different pathways and is processed along two 
time scales: one is a fast neural network (NN) and the other is a slow network 
called the glial network (GN). It was found that the neural network is 


powered and controlled by the glial network. Based on our biological 


knowledge of glial cells and the powerful concept of modularity, a novel 
approach called artificial neuroglial Network (ANGN) was designed and an 
algorithm based on different concepts of modularity was also developed. The 
implementation is based on the notion of multi-time scale systems. 
Validation is performed through an asynchronous machine (ASM) modeled 
in the standard singularly perturbed form. We apply the geometrical 
approach, based on Gerschgorin’s circle theorem (GCT), to separate the fast 
and slow variables, as well as the singular perturbation method (SPM) to 
determine the reduced models. This new architecture makes it possible to 
obtain smaller networks with less complexity and better performance. 
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1. INTRODUCTION 

The last few years have witnessed a tremendous growth in the field of intelligent systems. Inspired 
by biological neural networks, one such success has been achieved in evolution of artificial neural networks 
(ANNs). ANNs are characterized by their distinctive capabilities of exhibiting massive parallelism, 
generalization ability and being good function approximators. This renders them useful for solving a variety 
of problems in pattern recognition, prediction, optimization and associative memory [1], [2]. Additionally, 
they are also being employed in system modeling and control [3], [4]. 

These ANNs, efficient in numerous applications, are not as well suited for approximating non-linear 
and high-dimensional functions with multiple time dynamics like the ones in singular perturbation systems 
(SPS;) which increases the difficulties in system modeling, analysis and controller design .An effective way 
to overcome this problem is to separate the original system states into subsystems that change rapidly and 
those that vary slowly on the chosen time scale, using singularly perturbation method (SPM). 

Some recent research results using the (SPM) to analyze and control the SPSs are published in [5], 
[6]. However, accurate and faithful mathematical models for those systems are usually difficult to obtain due 
to the uncertainties and nonlinearities. In this case, adequate system identification becomes important and 
necessary, before a singular perturbation theory-based control scheme can be designed. 

Recently, research using multi times scale neural networks have been proposed in literature to solve 
the system identification problem of the nonlinear SPSs. Among them there are multi-time-scale dynamic 
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neural network (DNN) proposed in [7]. Or recurrent neural network (RNN) proposed in [8]. In these papers, 
training methods are based on a gradient descent updating algorithm with fixed “learning gain”, such as back 
propagation (BP) and RNN algorithms. The main drawback of these training methods is that the convergence 
speed is usually very slow. To accelerate the training process ,researchers investigated the extended Kalman 
filter (EKF) based training methods for NN in [9]. The theoretical analysis of EKF based training algorithm 
requires the modeling uncertainty of the NN to be a Gaussian process, which may not be true in real 
applications. Some other researchers also studied optimal bounded ellipsoid (OBE) algorithm-based learning 
laws for NN [10]-[12]. All of these methods are complex and computationally intensive. 

In this paper, we propose a new multi time-scale NN architecture called "artificial neuroglial 
network" (ANGN) based on the powerful concept of "modularity" to solve the problems of singular 
perturbation system training. The basic idea is to use the knowledge about the nervous system and the human 
brain, where the information flows along two different pathways and is processed along two-time scales: one 
is a fast-neural network (NN) and the other is a slow network called the glial network (GN). It was found that 
the neural network is powered and controlled by the glial network [13], [14]. 

In our experiment, for a given application, depending on the complexity and the physical 
characteristics of the problem, we divide our global model into two sub-models: slow and fast ones using the 
singular perturbation method (SPM). The first difficulty that arises when decoupling variables is the 
identification of both the slow and fast model variables. The solution is based on Gerschgorin’s circle 
geometric theorem (GCT) [15]. This technique makes it possible to locate the eigenvalues in the complex 
plane within groups of circles. The grouping of the modes is immediate whenever circles are disjoint, and 
afterward, the number of slow and fast modes is determined. 

Validation of the proposed approach is carried out on the ASM model, under the singularly 
perturbed standard form. Subsequently, an algorithm is adopted to test the effectiveness and performance of 
the proposed ANGN. This new architecture has made it possible to obtain networks of considerably smaller 
size with simple structures which have a strong nonlinear approximation capability and which enables it to 
model nonlinear singularly perturbed systems more accurately with less computation complexity, compared 
to the conventional neural network model. 


2. SINGULAR PERTURBATION METHOD 

This method is used for multi-time scale systems that can be reduced to the standard form of 
equation (3) by the determination of the parasitic term e. Consider the state model of a linear system of 
dimension n: 


og 0 


Evolving according to two-time scales, it can be decoupled into two slow and fast subsystems. The 


state vector X contains all the state variables corresponding to the dynamic elements. If x is the set of state 
variables of slow elements, and z is the set of fast elements the model is written 


Ela, Pte 
|= eer, 
K Ax, Anz B, (2) 


y=[c, ol" 


with : 
-X(to) = xo and z(to) = Zo 
A A,B Ap Aj, B, 


21>“ 22>- 2- are very large compared to 


* * * 
We introduce the parameter £ to normalize our model, we write: A»; = éA,, As, = éA,, B, = éB,. € can 


ee es 7 -1 
be given by: £ = laz lA] + lá . TA) with : Ly = -An “Ad ng 0 =A, + Aj, Ly 


By assuming that matrix A22 is invertible, the state equation in the standard singularly perturbed 
form with e as the perturbation parameter is then written as: 
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x| |Ay A Alte Bish 3) 
Z Azi An Z B; ? 


2.1. Slow and fast reduced models 
The slow reduced model is determined from eq. (3) by considering that € =O: 


x, = A,X, + Bu, 
Ys = Cx, + Du, (4) 
Zs = -A> (4x, + B,u,) 


where * s3 Z 5 JU s3 y s are the slow components of the variables ¥,2,U , Y respectively, with : 


A, =A\,— Ap AnA 
B, =B,- A, AB, 
Cc, =C,- CAD Ay 
D, = —C, AB, (5) 


with: x, (t 0 ) = Xo -The initial value of the slow components Z, is : Z, (t 0 ) = -A7 Asx: (t z) 
which is generally different from Z,. The fast variables Z cannot therefore be approximated by Z, in the 


time interval [o, T] . We introduce the corrective term Z ; , defined by : Z; = Z — Z, which represents rapid 


changes in Z .Therefore, the boundary layer equation, expressed in the dilated time T follows: 


dz , 7 dz 


(E = 
dt dt 


(6) 
The fast reduced model is then written: 


dz 
ae = AnrZ, (c)+ Buu, (x) (7) 
eS CZ; (z) 


Zr (¢,) = Zo + A3 AzXo 


3. IDENTIFICATION OF THE GEOMETRIC DYNAMICS 

Setting the previous standard form assumes: a) Knowledge of the eigenvalues to determine the size 
of the slow and fast eigenvectors. b) A suitable grouping of slow modes and fast modes. 

Our attention is focused on geometrical methods including the circles of gerschgorin. The 
localization of the eigenvalues on the complex plane makes it possible to put a system in the standard form 
without having to calculate these eigenvalues. In the case of GCT, the grouping of modes is immediate as 
soon as the eigenvalues are circumscribed in disjoint circles. The geometric method based on GCT for the 
selection and separation of different time scales is represented as follows: 


3.1. Gerschgorin’s circles theorem (GCT) 


Noting (a 1, f= hata n) as the elements of the state matrix A ,p;, Q; are expressed by 


ij? 
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P; => Ja, $= 12s wn % 


ener „n (9) 


Geometrical separation of the different dynamic modes is based on the application of the following 
two theorems. These theorems obtained from Gerschgorin give a localization of the eigenvalues on the 
complex plane. 


3.1.1. Theorem 1 

All the eigenvalues of a matrix of arbitrary rank n, are contained in n circle bundles centered at 
a11, Q22, «++, Annand radii Ry, Riz, .-.., Rin for the lines or Re1, Rez, ..., Renfor the columns, which are 
obtained by summing the modules of the off-diagonal terms appearing in the same line or column: 


Ry =p; and Rai = Qi. 


3.1.2. Theorem 2 

When a group of k line-circles (or k column-circles) is completely disjoint from the other circles, it 
contains k eigenvalues [16]. When a group of k circles is completely disjoint from the other circles, it can be 
said that the system then has at least two-time scales. Whether this group of circles is to the right or to the left 
of the other circles, we can determine the k slow modes corresponding to these k circles or respectively the k 
fast modes. Each circle represents a state of the system. It is then possible to give an adequate partition of the 
model. Generally, this direct method does not make it possible to conclude immediately in all cases. 
Dauphin-Tanguy [17] then proposes the use of transformations: 


S, = diag (,...1,@,,1...41),K=1, 2,......505 


these parameters allow the variation of the circles size and their optimization lead to circles of minimum 
radii. However, this method does not separate all systems. 


3.1.3. Changing the radius size 
Let the matrix be: 


Dadasi aia e ic i aa (10) 


The change of base X =S ¿X leads to a new state matrix. The radii R,, and R, become 


Rid, and R , / Œ, respectively [18]. If the operation is repeated several times, the aggregated 
transformation is: 
S=HS, 


X=SX with: k 
A=SAS" (11) 


If there are two disjoint sets of circles, then the permutation matrix is: 


F = PX 
=, -1 
A= PAP (12) 


3.1.4. Moving Circle Centers 
In order to improve the separation of dynamics, it is sometimes necessary to introduce a 
displacement of the circles which is characterized by the following transformation [18]: 
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T, =I +B, J;. (13) 


Only the elements of line Í and column j change, the centers of the circles i and j are shifted from 


danda to a; + Bia j anda ,, + Bia ;;, respectively. The choice of B can be made in such a way that 


X; =a;+B(a; -a;)-B;a, =( If several circles intersect, the terms B, (7 = 1,2,...)are calculated in the 


same way, so the final transformation is: 


ASTA with Ten., (14) 
A=TAT" 


If two groups of circles are disjoint, the permutation matrix P is again: 


o = PX 
= Si 
A= PAP (15) 


4. CONCEPT OF MODULARITY 
The application of concept of modularity to define the new architecture of small artificial networks 
involves the following four steps: 


4.1. The decomposition 

The decomposition of a task into subtasks is the first step toward the application of modularity. It 
can be done on the input space (horizontal decomposition) or on the input variables (vertical 
decomposition) [19]. 


4.2. Organization of the modular architecture. 

The interconnection of the modules can be parallel or in series. In the parallel architecture, all 
modules process their information simultaneously. The global output involves some modules or all of them, 
depending on the application. The cooperation link between the modules which can be of type "and" or of 
type "or" [20]. 


4.3. Nature of learning 
The organization of NNs in a modular architecture makes learning more difficult. The modules of 
such an architecture can follow different learning processes. 


4.3.1. Independent learning 

Training modules independently seems to be the simplest way. This suggests that the other modules 
of the architecture do not participate in learning. The interaction between the modules then occurs only 
during restitution phase [21]. 


4.3.2. Cooperative learning 
The proposed idea is to use a global method to train all modules at the same time. It is necessary 
then to have a fixed architecture, determined in advance. An example is given by ME [22]. 


4.4. Communication between modules 

The techniques for calculating the overall output of a multi-network architecture are diversely 
varied, among which is the technique of weighted votes [23]. A weight is then associated with each classifier 
representing a measure of performance. Another technique is to minimize the mean square error (MSE) of 
the global output. 


5. PROPESED NEUROGLIAL NETWORK ARCHITECTURE 
The architecture adopted for our ANGN is partly based on the concepts of modularity and remains 
very close to the above-mentioned architecture "ME". In this architecture, a number of NNs (experts) are 
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supervised by a GN (Figure 1). The glial supervisor network determines the weighting coefficients of each 
expert’s participation according to the input. The ANGN uses the "divide and conquer" strategy in which the 
responses from the expert NNs are combined into a single rapid response. The latter is aggregated to the slow 
response of the glial supervisor network, resulting in the overall response of our system. Algorithm 
developed in this article is based on softmax function. In this algorithm, the supervisor GN evaluates the 
performance of each expert NN according to the input and selects the best of them to be activated. 


Expert network 2 


Expert network k 


Figure 1. Neuroglial network architecture 


As illustrated in Figure 1, our global ANGN is composed of K fast NNs and a slow supervisor GN. 
The vector of the inputs is divided into two vectors X, and X; representing, respectively, the slow inputs and 
the fast inputs of the network. The vector X, is assigned to the supervisor network and the vector X; is 
assigned to the various experts. The responses of the expert modules are combined to form the fast output. 
The supervisor GN has two outputs, the first one, which is used for control and supervision of the experts by 
selecting the most suitable network and deactivating the others for each input vector. The second output 
represents the slow response of the GN, this output is aggregated at the fast output to form the global 
response of the ANGN. The structure of the global ANGN is close to multi-model approach. Indeed, each 
expert network is specialized in a precise sub-problem, a vertical decomposition of the input variables into 
fast and slow inputs, as well as a horizontal decomposition of the fast input space X; and slow X; is used. 


5.1. Algorithm based on function softmax 

In the ANGN, the vector of the fast inputs X;is sectioned both sequentially and in parallel into K 
vectors Xj, Xj2,...., Xx . These K vectors X constitute the respective inputs of the K experts. The slow input 
vector Xsis also sectioned in the same way into K vectors Xs;, Xs2,..., Xs which are applied consecutively to 
the supervisor GN. 

In this algorithm, each vector of the fast inputs xjf), (i = 1,2,...,K) is applied to all experts at the 
same time. These modules learn different examples from the learning base and specialize in specific groups 
of responses that are then weighted by the supervisor GN according to their absolute differences and the 
desired response. The expert whose response is the closest to the desired response will have the 
highest weight. 

The supervisor GN, whose input is the vector of slow inputs x;,, evaluates the performance of each 
expert according to the input and selects the best one to be activated. This algorithm shows many similarities 
to the one developed by Jacob [22] in the architecture "mixtures of experts". These similarities are related to 
the supervision and selection of experts, however there are three main differences: 

e Inthe "mixture of experts" architecture, the supervisor and the experts have the same time scale. For our 
ANGN, the two types of modules work on two different time scales, slow and fast. 

e The task of the supervisor in the ME approach is to supervise and control the competition of experts. In 
our approach, the GN supervises and controls the competition of experts as well as contributing to the 
overall response of the system by providing the slow response. 
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For the selection, the experts' weighting is binary (0 or 1). In the ME approach weights are probabilities 
between 0 and 1. 
Before presenting our algorithm, it is worth noting that the experts’ learning are cooperative, which 


means experts learn simultaneously and divide the task during the learning process. The weights of the expert 
and those of the GN, for the selected vector, are updated at the same time by propagation. The learning of 
experts and the GN is carried out simultaneously by following these steps: 


6. 


1. The separation of input vector X into both slow and fast vectors: x, and xp. 
2. Each vector xj), (i = 1,2,...,K) is intended for all experts. 

3. 
4 
5 


Each vector x;5), (i = 1,2,...,K) is intended for the supervisor GN. 


. The learning of the GN to obtain the desired slow response corresponding to the input. 
. The selection of the expert according to the value of the probability p(i/x,) and selection of the ith expert 


by evaluating the slow inputx,, . 
The output of expert i represents the conditional average of the desired response with respect the input 
and the expert network. 


The learning algorithm of the ANGN architecture. 


1. 
2. 


Initialization of the synaptic weights of experts and the GN. 
for each slow input vector x;5) : 
2.1 Calculate for i = 1,2,..k 

form = 1,2,...,q 


u, (k)=X i.) a,(k) 


oe SP) 
Deexp(u ; (k)) 


m T m 
Yin =X icp) Wiep ®) 


Yp (ELY Yi >> yi] 
—] 2 
gode "fa, — Yip | ) 


i k = 2 
28; ef= a, a Y| ) 
Yi» = X to Wio (k) 


exo (k) = a3" (k)— yro (k) 


Wie) (k + 1) 5 Wilo (k)+ eis) (K)X io 


er) (k) 5 das — YK (x) 
Wiin (k +1)= Wil (kd + nh, (k ej, (KX 
a,(k ae 1) z a,(k)+ mh,(k)— Blk) Xa. 
end for 

2.2. Repeat step 2.1 until the algorithm converges. 

2.3. If max (g,) than g, =1 


else g; =O 


if) 


K 
Yig) F Deus f 
2.4. i=l Aggregation of the two slow and fast outputs 


K 
Y D Ow Yi 
i=1 


end for 
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6. APPLICATION 

The ASM is a highly coupled nonlinear complex system and is a typical example of two time-scales 
system. The performance of the proposed ANGN architecture is assessed on both the reduced, slow and fast 
models of the machine. The state model of the induction machine in the stationary coordinate system (a, f) 
can be written as: 


I “_R(-@ 
d Prap S oT, i oT, ( ) | Poon +f er] (16) 
dt | Drap l-0 ( ) 1 I Prap 0 
oT,B, oT, * 
Lon -= PĒ- oL, ROJ, rap (17) 
oL, l 
Or: 
1 B 
T, L, J R(- 8) 
“ll-o 1 
TB RQ) a e 


with :T,, =of,, T_,=oT, B,= ando = (1 — M?)/(L; * L,) ) 


Application of the GCT to the state matrix A results in two circles which intersect (Figure 2a). A 
change in the size of rays is carried out by transformation 1. 


— Prap — I, (0) 
asalga [e zeo 


To get: 
1 
= T I, mots 
A, —% sp sp 
1— 1 ` 
Zr IL,- oJ, (18) 
Tis Tz 


The centers of the two circles (Figure 2a) are relocated by: 


h =R P= a 


—T, I, 
And 
1 
(0) — I, 
Tp 
A, = 
1 
= 2O T; pal; SA; oJ, 


The new line circles are still double and disjoint (Figure 2b); the final transformation is: 


Prap I O 
= ; =" 2 (19) 
. pege] ne |- I, B,R(- 0) 
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100 = < ee eee 
p SS is 100 > = 
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= o = = of 
-50 `~ < i g 
ae i | ge -100 = - 
-100 Di 1 = mal ~ 
-150 -200 
-200 
E È E -300 E 2 
-250 -200 -150 -100 -50 oO -200 -100 oO 100 
Re Re 
(a) (b) 


Figure 2. (a) Circles that intersect; (b) Disjointed Circles 


In this case, the slow and fast components are easilyidentified. We can then apply SPM to develop 
the slow and fast submodels. By putting: 


We get: 
0 ee 
d XxX _ Ty | X $ L, O . Vrap ; (20) 
dt ly æ+l; lz] [-73 B RCO)j| 0 
2 2 
Iy Tp 
Lo = Laz 
L, 


This form is standard, the flux x is slow and the flux z is fast. By decomposing the fluxes, the voltages and 
the torque, we obtain: 

x= x,(r) 

z= z,()+2/() 

V sap = Vecp(s)\t) + Vasile) (21) 

T =l aTh (f) 


em em 


The reduced slow model is then: 


1 
d Psa = (1 + ar. . Psa a . Vsa(s) 
Psp d| o _ 1 Pp l+a@ 
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The reduced fast model is: 


l+a 


— — O 
ae = T, [re |-£ ee 
dt | Lay) o ita || Isao] L [Yo 


1 ( ao T 
I =) +—P yp ; 
sap(s) Ly f +a Øsapls) l+a ict) 


Tons) = Pal pts) Pop! salt) 


1 ao L 
Prap(s) = A ro(ı = EZ Jhen = a) 


L, 
rapt) =~ B ROW apts) 


6.1. Results and discussion. 

The ANGN_MAS model is composed of two groups of inputs Vsa, vsg, which are decomposed into 
two slow inputs Vsas), Vsp) and two fast inputs Vvsag), vg. The ANGN_MAS consists of four experts and a 
glial supervisor network. The four experts and the glial network have similar architectures, consisting of an 
input layer of four neurons and an output layer of one neuron. The effectiveness of the ANGN algorithm 


proposed in this paper is illustrated by the performance index root mean square (RMS) value. The RMS of 
the states error is calculated as: 


age | CPO), 


where n is the number of simulation steps, and e(i) is the difference between the state variables of the model 
and the true system at the ith step. Figure 3 (a) and (b) show the good convergence of this learning algorithm. 
The minimum value, which is very close to zero occurs after only two iterations. The algorithm converged 
perfectly and quickly. 


x 107° Pe: 10°° 
— ~ Expert 
— Expert2 
0.8 0.8 ——— Experts 
—_———— Expert4 
0.6 0.6 
(Jp) a 
= = 
0.4 0.4 
0.2 Deed m 0.2 
Y: 7-55e-030 
=æ 
5 L | ó L L 
2 4 2 4 
iteration iteration 
(a) (b) 


Figure 3. (a) Evolution of the glial network; (b) Evolution of experts 


For a comparison with the algorithms in [7], we choose the same parameters of the induction motor. 
The simulation results are presented in Figure 4 and 5. The RMS values for state variables are presented in 
Table 1. 
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b 4 b 3 b b 


Mg 2 Avg 


Aig 


Aw, (WD) 
> ~n 
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Sna 
Aia (A) 
o> 
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o 


-2 


-2 


0 of 02 03 04 05 o1 o2 03 04 05 0 M 02 03 04 05 997 o2 o3 04 05 
Time (second) Time second Time (second) Time (second) 
Figure 5. Error for states P,a, Prg sta, ig: [7] 
Table 1. The RMS values for state variables 
Pra(wb) rg (wb) ina) iga) 
RMS (ref [7]) 0.0575 0.05334 0.0446 0.0452 
RMS (ANGN) 1.029*1075 3.33*1074 3.33*1079 9.8*1073 


The RMS values of all state variables in Figure 4 and 5 demonstrate that the performance has been 
improved compared to those [7], it is very clear from these figures that the identification errors 
AQq , App, Aig , Aig of the proposed ANGN are greatly reduced.. We can confirm that the state variables of 
ANGN follow those of the nonlinear system more accurately and faster. This is due to the good separation of 
slow and fast modes which result that the complexity of the architecture sub-networks of ANGN are 
considerably reduced. 

We conducted other comparisons of the proposed approach with both the modular approach in 
[24] and the mixture of experts (ME) approach. The RMS values of the T,,, are given in Table 2. From Table 
2, it is very clear that RMS values of ANGN are much smaller than both the values [24] and ME, which 
means that the proposed algorithm in this paper can achieve more accurate results. From Figure 3(a) and 
3(b), we can see. Therefore the excellent performance of the proposed model in terms of convergence speed 
and reduction of the optimal RMS error than both the performance in Figure 6 and performance in Figure 7. 
It is largely due to the small size of the sub-networks and the very limited number of examples presented at 
the input of each expert. These advantages are the main properties of modularity. 
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Table 2. Comparison performance models 


Models Number of iterations RMS 
ref [24] 100 1.08932 E-27 
ME 90 7.934 E-24 
ANGN 2 17.455 E-30 
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Figure 6. Evolution of RMS of the ME model Figure 7. Evolution of RMS in ref [24] 


7. CONCLUSION 

In this paper, a novel architecture called artificial neuroglial network (ANGN), inspired by a 
recently discovered fact in biology, was developed along with a modular algorithm based on the softmax 
function. Because of its good characteristics, an ASM put in standard singularly perturbed form is chosen to 
validate the proposed ANGN architecture. The slow and fast reduced models are obtained in two steps: 
- Application of a geometric approach based on GCT to decouple the slow and fast variables. 
- Application of the SPM to develop the reduced models. 

The simulation results related to the learning of the ANGN_ASM model demonstrate the fast and 
accurate convergence properties of the proposed algorithm. ANGN can modeling ASM very wel with less 
computational complexity. The next step is to use these reduced models to develop controllers for the ASM. 
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